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[57] ABSTRACT 

A method for fusing two or more source images to form 
a composite image with extended information content 
and apparatus for forming the composite image from 
the source images is disclosed. Each source image is 
decomposed into a number of source images of varying 
resolution. The decomposed source images are ana- 
lyzed using directionally sensitive operators to generate 
a set of oriented basis functions characteristic of the 
information content of the original images. The ori- 
ented basis functions for the composite image are then 
selected from those of the different source images and 
the inverse of the decomposition performed to con- 
struct the composite image. Apparatus for fusing two or 
more source images to form a composite image is also 
disclosed. 
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dosed a means for implementing the pyramid process 

METHOD FOR FUSING IMAGES AND for the analysis of images. 

APPARATUS THEREFOR The Laplacian pyramid approach to image, fjision k 

perhaps the best known pattern-selective method. P. 
The invention relates to a method for fusing two or 5 Burt in Multiresolution Image Processing and Aiialys^ 
more source images to form a composite image with A. Rosenfeld, Ed., Springer Verlag. New York 1984 
extended information content and apparatus for form- first disclosed the use ofunage fus.on techmques ; based 
mgthecompositeimagefromthesourceimage, ^^St^.t^o^^ 
BACKGROUND OF THE INVENTION 10 closed the use of the Laplacian technique for the con- 
Image fusion is a process that combines two or more struction of an image ™* " °[ JJJ 
sourceWs to form a single composite image with ^^^SSSST. S fc SStS^SSS 
extended information content Typically images from cWterent tocaj semngs. a. iow m m^udc J ™" 

computer aided tomography (CAT) and magneac reso- * ™ IR iniageVfor surveillance applica- 

nance imaging (MRI) systems, are combined to form como * Proceedings of 

the composite image. Multiple images of a given scene ^ c^^cc on Computing in Aerospace, 
taken with different types of sensors, such as visibkand ^ October 1991 have disclosed a 

infra-red cameras, or images ; taken with a given type of ^ ^ d for camm with 

sensor and scene but under different imaging condition, graphically generated imagery as an aid to aircraft land- 
such as with different scene illumination or camera £ K Burt and Adelson m ACM Trans, on Graphics., 
focus may be combined. Image fusion is successful to volume % ges 217-236 (1983) and in the Proceeding 
the extent that: (1) the composite image retains all useful Qf spm volumc 575f pagcs 173-181 (1985) have devel- 
information from the source images, (2) the composite ^ rclated Laplacian pyramid techniques to merge 
image does not contain any artifacts generated by the mtQ mosaics for a variety of applications 

fusion process, and (3) the composite image looks natu- ^ & i^ p \ &cim transform is used to decompose 
ral, so that it can be readily interpreted through normal eacfa mto reg ular arrays of Gaussian-like 

visual perception by humans or machines. The term basis functions 0 f many sizes. These patterns are some- 
useful information as deterrnined by the user of the ^ re f errea * to & basis functions of the pyramid trans- 
composite image determines which features of the dif- fonnj or ^ wavelets. The multiresolution pyramid of 
ferent source images are selected for inclusion in the sq^cc images permits coarse features to be analyzed at 
composite image. l ow resolution and fine features to be analyzed at high 

The most direct approach to fusion is to align the resolution. Each sample value of a pyramid represents 
source images, then sum, or average, across images at 35 ^ e amplitude associated with a corresponding basis 
each pixel position. This and other pixel-based ap- function. In the Laplacian pyramid approach to fusion 
proaches often yield unsatisfactory results since individ- above, the combination process selects the most 

ual source features appear in the composite with re- prominent of these patterns from the source images for 
duced contrast or appear jumbled as in a photographic inclusion in the fused image. The source pyramids are 
double exposure. 40 combined through selection on a sample by sample basis 

Pattern selective image fusion tries to overcome these to form a composite pyramid. Current practice is to use 
deficiencies by identifying salient features in the source a "choose max rule" in this selection; that is, at each 
images and preserving these features in the composite at sample location in the pyramid source image, the source 
full contrast Each source image is first decomposed image sample with the largest value is copied to become 
into a set of primitive pattern elements. A set of pattern 45 the corresponding sample in the composite pyramid, 
elements for the composite image is then assembled by Finally, the composite image is recovered from the 
selecting salient patterns from the primitive pattern composite pyramid through an inverse Laplacian trans- 
elements of the source images. Finally, the composite form. 

image is constructed from its set of primitive pattern In the case of the Laplacian transform, the compo- 
elements. 50 nent patterns take the form of circularly symmetric 

Burt in 'Vast Filter Transforms For Image Process- Gaussian-like intensity functions. Component patterns 
ing", in Multiresolution Image Processing And Analy- of a given scale tend to have large amplitude where 
sis, volume 16, pages 20-51, 1981 and Anderson et al in there are distinctive features in the image of about that 
U.S. Pat No. 4,692,806 entitled "Image-Data Reduc- scale. Most image patterns can be described as being 
tion Technique", incorporated herein by reference for 55 made up of edge-like primitives. The edges in turn are 
its teaching on image decomposition technique, have represented within the pyramid by collections of corn- 
disclosed an image decomposition technique in which ponents patterns. 

an original comparatively high-resolution image com- While the Laplacian pyramid technique has been 
prised of a first number of pixels is processed to derive found to provide good results, sometimes visiblearti- 
a wide field-of-view, low resolution image comprised of 60 facts are introduced into the composite image. These 
second number of pixels smaller than the first given may occur, for example, along extended contours m the 
number. The process for decomposing the image to scene due to the fact that such higher level patterns are 
produce lower resolution images is typically performed represented in the Laplacian pyramid rather indirectty. 
using a plurality of low-pass filters of differing band- An intensity edge is represented in the Laplacian 
width having a Gaussian roll-off. van der Wal in U.S. 65 pyramid by Gaussian patterns at all scales with positive 
Pat No. 4,703,514 entitled "Programmed Implementa- values on the lighter side of the edge, negative values on 
tion Of Real-Time Multiresolution Signal Processing the darker, and zero at the location of the edge itself. If 
Apparatus", incorporated herein by reference, has dis- not all of these primitives survive the selection process, 
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the contour is not completely rendered in the compos- FIG. 9 is a timing diagram of when the various un- 
ite. An additional shortcoming is due to the fact that the ages and pyramid levels may be computed in a system 
Ome^^eo^o^'^tt^l^'f&a^ mean 5 : with I/O frame stores, and assuming interlacel/O. 
values. Errors in the selection process lead to changes in FIG. 10(a) is a photograph of a source image from a 

the average image intensity within local regions of a 5 sta fL d ^ d ^ ble >>ght "mera. 

scene. These artifacts are particularly noticeable when . ™3. 10(6) is a photograph of a source unage from an 

sequences of composite or fused images are displayed. ""ff? 1 ^ . . . ... . „„, • u 

s^sssxs: - -jut."--*-— 

because of noise in the image or sensor motion, the 

selection process may alternately select the basis tunc- DETAILED DESCRIPTION 

tions from different source images. This leads to unduly A flow chart for a prior ^ pattern-based image fo- 
perceptible artifacts such as flicker and crawlers. J5 sion ^ shown ^ pr Gi t The source images are assumed 
Thus there is a need for improved methods of image to ^ prior to undertaking the fusion steps. The 

fusion which overcome these shortconrings in the prior fasj on methCK j comprises the steps of ti^forming each 
art and provide better image quality in a composite source image I„ into a feature-based representation 
image formed by the image fusion process, particularly wnere each image I„ is decomposed into a set of compo- 
when sequences of composite images are displayed. 2 0 nent patterns Pn(m), where n= 1, 2, ♦ . . , N, the number 

SUMMARY OF THE INVENTION of ™»* m =*'*> • • • » M the number of 

patterns in the set for the n rt source image. Features 

A method of the invention for forming a composite from the source images are combined to form a set of 
image from N source images where N is greater than component patterns Pc(m) representing the composite 
one comprising the steps of decomposing each source 25 image assembled from patterns in the source image 
image In into a plurality L of sets of oriented compo- pattern sets. The composite image Icis then constructed 
nent patterns PMm, L); computing a saliency measure from its component patterns Pc(m). 
SMm, L) for each component pattern PM^i L ); select- The Laplacian pyramid method for image fusion can 
ing component patterns from the component pattern be described in this framework. Performing the Lapla- 
sets P/v(m, L) using the saliency measures S/Xm, L) for 30 cian transform serves to decompose each source image 
each component pattern to form a set of oriented com- into a set of circularly symmetric Gaussian-like compo- 
ponent patterns Pc(m, L) of the composite image; and nent patterns. The pyramid is a regular decomposition 
constructing the composite image from the set of ori- into a fixed set of components. This set consists of pat- 
ented component pattens Pcdn, L). terns at different scales, represented by the pyramid 

The invention is also an apparatus for forming a com- 35 levels, and different positions in the image, represented 
posite image from a plurality of source images compris- by the sample positions within the pyramid levels. Let 
ing means for decomposing each source image into a P*0£k) be the Laplacian value at location ftj) m 
plurality of sets of oriented component patterns; means pyramid level k for image n. This value represents the 
for computing a saliency measure for each component amplitude of a corresponding component pattern 
pattern; means for selecting component patterns from 40 PndJ.k) which is a Russian-like function. 
^ ^ . i- A flow chart for the seneration of the Gaussian and 

the component pattern sets using the saliency measures 1 . cnari .] ~f KC c a „,_„~r"l~, 

for each^compo^ent pattern to form a set of oriented ^ 

component patterns of the composite image; and means ^ k ~Kc& ' by Flj a low ^ filter having a 
for constructing the composite unage from the set of ^ ^ y J^ kd P by F2> to rcmo % e 

onented component patterns alternate pixels in each row and alternate rows, to form 

BRIEF DESCRIPTION OF THE DRAWING the first level Gaussian G(l). The lower level Gaussians 

™~ * - r, v , ■ X* G(n) are formed successively in the same way. The 

FIG 1 is a flow chart showing a prior art method for L(n) corresponding to the Gaussian at each 

pattern-based image fusion. x levd of ^ pyramid ^ formcd by restoring the subsam- 

FIG. 2 diagrammatically illustrates a method for pled ^ to the next lowest level of the Gaussian 
forming the Gaussian and Laplacian pyramids. pyramid (by inserting zero-valued samples between the 

FIG. 3 diagrammatically illustrates a method for ^ yca ^ p2' then applying an interpolation filter 
reconstructing the original image from the Laplacian F1 j ^ subtracting from the Gaussian of the given 
pyramid. 55 level. The Laplacian formed in this way is known as the 

FIG. 4 diagrammatically illustrates a method for Reduce-Expand (RE) Laplacian. Alternatively, the 
pattern-based image fusion of the invention. Laplacian can be formed, without subsampling and rein- 

FIG. 5 illustrates the implementation of the method terpolation as shown by the dotted line FIG. 2. This is 
of the invention in real-time digital image processing called a filter-subtract-decimate (FSD) Laplacian. In 
hardware. 60 FIG. 3 a method for reconstructing an image from the 

FIG. 6 is a schematic circuit diagram of the circuits Laplacian pyramid is illustrated. In this method the 
P5 and P6. Laplacians are interpolated and summed to reproduce 

FIG. 7 is a schematic circuit diagram of the circuit the original image (i.e. the inverse RE Laplacian 
P7. pyramid transform). 

FIGS. 8(fl), (c) and (</) are schematic diagrams of 65 The step of combining component patterns, FIG. 1, 
circuits implementing the weighting function. uses the choose max rule; that is, the pyramid con- 

FIG. %{b) is graphical illustration of a particular structed for the composite image is formed on a sample 
weighting function. basis from the source image Laplacian values: 
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been found to provide excellent results even where 
k)=mMx [LMk) t L*iJ,k) Lr&iM image fusion based on the Laplacian pyramid intro- 

* ^^^-^^^ artifacts^ r - .v. ! ^,:> ■ * - 

where the function max [ ] takes the value of that one of Several known oriented image transforms satisfy the 

its arguments that has the maximum absolute value. The 5 requirement that the component patterns be oriented 

composite image If is recovered from its Laplacian ^ havc mean> The gradient pyramid has basis 

pyramid representation P c through an inverse pyramid functions of many sizes but, unlike the Laplacian 

transform such as that disclosed by Burt in "Fast Filter pyram id f these are oriented and have zero mean. The 

Transforms For Image Processing", in Multiresolution ^gnt pyramid's set of component patterns P„(m) can 

Image Processing And Analysis, volume 16, pages , 0 ^ rcprescilted ^ p^ij.k.1) where k indicates the 

20-51, 1981 and Anderson et al in U.S. Pat. No. pyrar ^ d le vcl (or scale), 1 indicates the orientation, and 

4,692,806. . y the index position in the k,l array. The gradient 

A method of the mvennon for forming a composite £ value D^jW) is the amplitude associated 

image from a plurality of source images, as shown in 3ernP ftYktt It can be shown that the 

images into a feature-based repre^nUtion by ^ t ^ f . Ga ^Sbasl functions of many scales and ori- 

££r^ entations. One such basis function J-"*^ 

Kiting a salienc? measure for each component pat- each sample in the pyramid When the^are scaled m 

S cabining the salient features from the source m amplitude by the sample value, and summe^ the <^n^ 

SeTS assembling patterns from the source image 20 nal image is recovered exactly, ^^s^manon 

pattern sets P„(m) guided by the saliency measures are implicit in the inverse pyramid transform.1t is to be 

S n (m) associated with the various source images; and understood that oriented operators other than the gradi- 

constructing the composite image through an inverse ent can be used, including higher derivative operators, 

transform from its component patterns P<(m). A sa- 2J and that the operator can be applied to image features 

liency estimation process is applied individually to each other than amplitude. 

set of component patterns P„(m) to determine a saliency An alternative way of analyzing images is to use 
measure S„(m) for each pattern. In general, saliency can wavelet image representations. Wavelet image rcpre- 
be based directly on image data, I„, and/or on the com- sentations, as disclosed for example by Rioul et al in an 
ponent pattern representation P„(m) and/or it can take article entitled "Wavelets And Signal Processing" in 
into account information from other sources. The sa- the IEEE Signal Processing Magazine, October, 1991, 
liency measures may relate to perceptual distinctiveness pages 14-38, are oriented spatial functions, linear com- 
of features in the source images, or to other criteria binations of which can be used to define an image. In 
specific to the application for which fusion is being the 0 f a wavelet representation, there are at least 
performed (eg., targets of interest in surveillance). two sets of wavelets for different orientation. Typically 
The invention is a pattern selective method image fixee. sets of wavelet basis functions, a set of horizon- 
fusion based upon the use of oriented functions (compo- tally oriented functions, a set of vertically oriented 
nent patterns) to represent the image and, preferably, an functions, and a linear combination functions derived 
oriented pyramid approach that overcomes the short- f rom W avelets having right and left diagonal orienta- 
comings in the prior art and provides significantly en- ^ Qn ^ ^ 0 f oriented basis functions which 
hanced performance. Each source image is, preferably, define t h e source images are obtained, a set of oriented 
decomposed into a plurality of images I of different ba&is functions for the composite is selected in the same 
resolution (the pyramid of images) and then decompos- way ^ for me basis functions generated using the gradi- 
ing each of these images into a plurality of sets of on- ^ operators ^ the composite image is then recon- 
ented component patterns. The oriented component structed from them. 

patterns are, preferably edge-like pattern elements of « ^ gradicnt pyram id for image I is obtained by ap- 

many scales and orientations using the oriented plying gradient operators to each level of its Gaussian 

pyramid. The use of the oriented pyramid improves the G / n) & described in Appendix 1. Four such 

retention of edge-like source image patterns in the com- l L ndicDts are used for the horizontal, vertical, and or- 

posite image. A pyramid is used that has component ^ogonal directions in the images, respec- 
patterns with zero (or near zero) mean value. This en- »> ^ * ^ four &Bdieais m then fused using a selec- 

sures that artifacts due to spurious inclusion or exclu- ^ ^ n sa]icacy to ^t the components 

sion of component patterns are not unduly visible. Com- to ^ ^ to form the gradient pyramid representation 

ponent patterns are, preferably, combmed through a ^ mposite image. To reconstruct the composite 

weighted average rather than a simple selection pro- £tne Sent pyramid representation?!!* 

cess. The most prominent of these patterns « >*cfed 55 ^^^^ ^ ^ to form 

for inclusion m the composite image at oefaj scale and j^orieff iec^ d ^ ative pyramids. These are 

orientation. A local sahency analysis wte>toqr at each level of the pyrainid tofb™ a standard 

may be based on Ae local edge energy (or overtask- jumrne^ thc coinpo>ite ^ h 

component ^^^^T^ information that is 

S^^^ ^S^ornT use^mmterpre^ 

^ ap^oach overcome^ artifacts that have been depend on the purpose for c^tructmg the 1 composite 
obs^eS m S-based fusion and in pattern-selective 65 image and any measure of sahency will be task depen- 

S^toTl^an pyramid. Weights are ob- dent. However, sahency generally increases with . the 

3 nTZoLZ sigmofd function of the sahency amplitude of the elementary pattern, Le . SjJmW) be 

measures. Image fusion using the gradient pyramid has the saliency value corresponding to P„(ij,k4). A sa- 
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liency measure that increases with the prominence of a 
component pattern can be indicated by its amplitude 



where 



Here D ft 0j,k,l) is the amplitude associated with the 
pattern P„(ijW) at position (ij) of gradient pyramid 
level k and orientation 1. Alternatively, it can be indi- 
cated by the prominence of that component and other 
components within a local neighborhood. This neigh- 
borhood is indicated by a weighting function w(i' j*): 



5 is the normalized saliency at the (i j) position, 1'* orienta- 
tion of the k th pyramid level for the n rA source image. 

This sigmoid like function accentuates the difference 
between weights of elements that have nearly average 
saliency while fixing the weights for a given element at 
10 near zero or near one if its salience is significantly below 
or above average, respectively. 

The final step in forming the composite image I* is its 
reconstruction from its gradient pyramid representation 
P 0 The details of the computation of the inverse gradi- 
Typically the neighborhood used are the component | 5 ^ pyramid transform are given in Appendix 1. 
patterns for the 3 X 3 array of nearest components to the The invention is also apparatus for forming a compos- 
particular component of interest or the 3 X 3 array of jte [ m ^ P f rom a plurality of source images comprising 
picture elements surrounding the picture element of means for transforming the source images into a feature- 
interest, depending upon the way the components are based representation by decomposing each source 
indexed. 20 image I tt into a set of component patterns P*(m) using a 

Another alternative measure bases salience on the plurality of oriented functions; means for computing a 
occurrence of specific patterns, such as targets in the saliency measure for each component pattern; means 
image. For example, S may be related to correlation of for forming the component patterns Pe(m) of the com- 
the source image with a filter matched to the target posite image by assembling patterns from the source 
pattern at each sample position. 25 image pattern sets Pn(m) guided by the saliency mea- 

sures S n (m) associated with the various source images; 



The gradient pyramid for the composite image Ic is 
obtained by selecting components from the source 
pyramid basis functions P tt for each set of oriented func- 
tions. Selection is repeated at each sample position 
based on the saliency measure. The selection rule com- 30 
monly used in current practice is "choose max", that is, 
select that source image sample that has the greatest 
amplitude. However a "soft switch" is preferable to 
strict selection; that is, when selection is between two 
component patterns that have quite different saliency, 35 
then the one with the larger saliency is chosen, but 
when selection is between components that have com- 
parable saliency, then the composite sample value is 
taken to be the weighted average of the source samples. 

The combination process is then one in which the 40 
amplitude of the combined pattern element is computed 
as a weighted average of the amplitudes of the source 
pattern elements for each orientation 1. 



45 



The weights used in this average are based on relative 
saliency measures over the source image. Weights are 
defined such that image components with higher sa- 
liency get disproportionately higher weight As an ex- 50 
ample, let A be the total saliency at a given position 



where N is the number of source images. 

For appropriately selected constants a and b, 0<a<- 
b<l,let 

ifT„<a 

for a<T„<b 

forb<T„ 



55 



60 



and means for constructing the composite image 
through an inverse transform from its component pat- 
terns Pc(m). 

Apparatus for implementing the method of the inven- 
tion is shown in FIGS. 5-8. The apparatus is shown in 
terms of two source images but it is understood that any 
number of source images can be used with appropriate 
modification of the apparatus. 

The frame stores FS1 and FS2, if necessary, are used 
to convert input source images generated in an inter- 
laced format to a progressive scan format for subse- 
quent processing and to adjust timing. A television 
camera output is typically in interlaced format. 

The combination of pyramid circuit PI and frame 
store FS3 are used to compute the k-level Gaussian 
pyramid representation G«(k) of the input source image 
la and the combination of circuit P2 and frame store 
FS4 are used to compute the n-level Gaussian pyramid 
representation GaOO of the input source image I*. The 
circuits PI and P2 provide the low pass filter with a 
Gaussian rolloff and the pixel subsampling (removal/- 
decimation of alternate pixels in each row and each row 
of the filtered image). 

The next operation on each level of the Gaussian 
pyramids G(k) is a filter (1 + wO which is performed by 
circuit P3 and circuit P4 to form G/(k) and G(k)/, 
respectively. The purpose of this pre-fUter P3 and post- 
filter P8 are to adjust overall filter characteristics to 
provide an exact correspondence between intermediate 
results in the gradient pyramid transform and the Lapla- 
cian transform. Alternatively, this filter may be applied 
at other points in the sequence of transform steps. Other 
similar filters can be used to obtain approximate results, 
w' is a three by three binomial filter: 



65 



W « (1/16) 



1 2 1 

2 4 2 

1 2 1 



And the filter P3 has the form: 
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1 + W - (1/16) 



V 2 
2 20 
1 2 



Next, each of the filtered Gaussian pyramids G/(k) 
and G/(k) is filtered with four oriented gradient filters 
representing the horizontal da, vertical d„ right diago- 
nal dnA and left diagonal dw filters respectively. 

l| 

*-n 



4r- 



1 0 
0 -1 



Id/ 



MT) 



These operations are performed by circuits P5 and P6, 
producing the eight oriented gradient pyramids 
DflOUi), Do(k ( rd), DaOcv), Da(k,rd), Dft(k ( h), D*(k,rd), 
Dafov), DaOcrd). It is to be understood that while the 
gradient operators shown here use only the two nearest 
neighbor samples in the particular direction, a larger 
number of neighbors can be used in the gradient calcu- 
lation. 

In FIG. 6, circuits P5 and P6 comprise four subtract- 
ers 61, 62, 63 and 64. The input signal is connected 
directly to an input of subtracter 61 and through a sin- 
gle pixel delay 65 to the second input of subtracter 6L 
The output of subtracter 61 is d*. The input signal is 
connected directly to an input of subtracter 62 and 
through a single line delay 66 to the second input of 
subtracter 62. The output of subtracter 62 is d*. The 
input signal is connected through pixel delay 65 to an 
input of subtracter 63 and through line delay 66 to the 
second input of subtracter 63. The output of subtracter 
61 is d/rf. The input signal is connected directly to an 
input of subtracter 64 and through line delay 66 and 
pixel delay 65 to the second input of subtracter 64. The 
output of subtracter 61 is did. P5 and P6 can be imple- 
mented using a commercial Field Programmable Gate 
Array circuit (FPGA) such as XC3042 manufactured 
by Xilinx, Inc., San Jose, Calif. 95124, 

The fusion function combines two or more images 
into a composite image as shown schematically in FIG. 
8(a). Here the fusion function is computed on the four 
oriented gradient pyramids of the source images. It can 
also be applied to the Laplacian pyramid directly, but 
with less effectiveness. 

The functional dependence of W n on the total sa- 
lience A for source image l n a is shown in FIG. 8(6) for 
the case of two input images. The functions: 

if{S„aj,W)/A(ij,k,l)}<a 

for a<{S n (ij,k,l)/A(iJ,k4)}<b 



can be implemented with a single look-up-table (LUT) 
5 of size 64K x 8 if the input and output images are 8 bits 
as illustrated in FIG. 8(c). 

As examples, saliency may be based on absolute sam- 
ple value or on a local root mean square average where 

In FIG. 8(<Q an implementation of the local average 
method is shown. 
Subsequently, a weighted sum of the oriented gradi- 

15 ent pyramids are computed in each of the orientations 
separately, resulting in the four composite oriented 
gradient pyramids D«(Ui), D<(k,rd), DcO^v), D e (k^d). 

The composite oriented gradient pyramids Dcfch), 
D«(k,rd), D^Ocv), Dc&rd) are then filtered again with 

20 the same four oriented gradient filters d*. d* dn/and did 
and added to each other, resulting in the composite 
Laplacian pyramid Lc(k). This computation is per- 
formed by circuit P7. In FIG. 7, circuit P7 comprises 
four subtracters 71, 72, 73 and 74 and three adders 75, 

25 76 and 77- The input signal d* is connected directly to 
an input of subtractor 71 and through a single pixel 
delay 78 to the second input of subtractor 71. The input 
signal d v is connected directly to an input of subtractor 

72 and through a single line delay 79 to the second input 
30 of subtractor 72. The input signal d*f is connected 

through a single pixel delay 80 to an input of subtractor 

73 and through a single line delay 81 to the second input 
of subtractor 73. The input signal du is connected di- 
rectly to an input of subtractor 74 and through a single 

35 line delay 82 and single pixel delay 83 to the second 
input of subtractor 74. The output of subtracters 71 and 
72 are connected to the inputs of adder 75. The output 
of subtracters 73 and 74 are connected to the inputs of 
adder 76. The output of adder 75 and the output of 

40 adder 76 divided by two (2) are connected to the inputs 
of adder 77. The output of adder 77 goes to frame store 
FS5. P7 can be implemented using a commercial Field 
Programmable Gate Array circuit (FPGA) such as 
XC3042 manufactured by Xilinx, Inc., San Jose, Calif. 

45 95124. 

The composite Laplacian pyramid LcOO is equivalent 
to the FSD Laplacian pyramid disclosed by Anderson 
et al, U.S! Pat No. 4,692,806, and is stored in FS5. The 
pyramid Le(k) is then filtered by circuit P8 to produce 
50 LF<(k). The filter P8 has the form (1 +w) where w is a 
five by five binomial filter: 



55 



w « WW - (1/256) 



1 4 6 4 1 

4 16 24 16 4 

6 24 36 24 6 

4 16 24 16 4 

1 4 6 4 1 



60 Notice that the (1 /V2) factor in the diagonal gradient 
filters can be combined into a single (1/2) factor in the 
diagonal components in the P7 processing function as 
shown in FIG. 5. 
The composite image I? is reconstructed from the 

65 composite RE Laplacian pyramid LFcOO using the 
combination of circuit P9 and frame store FS6 using the 
method described by Burt "Fast Filter Transforms For 
Image Processing", in Multiresolution Image Process- 
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ing And Analysis, volume 16, pages 20-51, 1981 begin- two pixels to the right of the edge pixel of interest are 
ning with the lowest level of the pyramid In FIG. 6 the substituted for the missing pixels, 
circuit P9 is the same as PI ancTPZ Th£ process starts " The construction 1 and processing as described^ here- 
with the remainder G<<k), and computes the next higher can use a processor per function and per pyramid level, 
resolution level Ge(k— I) using the following function: 5 where the processors at each subsequent lower ^resolu- 
rcwjuuuii i C « wcv / e e tion operate at half the clock rate. A significantly more 

c^-i)=ii ? < (*-i)+w*O e , <A) efficient implementation is to use one processor per 

function, where each processor computes or processes 
as shown in FIG. 4 where G c «(k) is G<(k) expanded to all pyramid levels in sequence as disclosed by van der 
the next pyramid resolution level by inserting zero val- 10 Wal in U.S. patent application Ser. No. 07/805149, 
ued samples between the given samples and filtering referred to above and incorporated herein by reference, 
with w The result G<<k- 1) is stored in FS6. The next This can be accomplished by using flexible frame stores 
higher resolution level G«(k-2) is then computed in the (FS3, FS4 and FS5) for storing intermediate results, and 
sanie manner, combining G,<-i(k-l) and Ufk-2). adjusting the timing and/or processing clock * w*om- 
This process is repeated until GAP) has been computed 15 modate the additional processing time required. The 
and stored in FS7. FS7 is used to adjust timing and to processing of all pyramid levels by a single processor 
convert the composite image from progressive scan to typically requires 4/3 tunes the number of operations 
n interlace scan format if It is required by the image required for processing one full resoluUon^ By 

, „ #A . MMM , using the blanking time of the input image efficiently, 

pass (level 0) the 20 the Ltual clock rite does not have, to be mcre^dby 
souL images GJ0) andG^O) from the input frame much, if at all, to accommodate the increase m required 
stores FS1 and FS2, func- P^mg. q{ ^ ^ 

rml^d £5? ZTZt Se G^rdSoS framestoreisshowninHG. 5. These are FS1, FS2,and 
G<f»° and G6(0). At ^^ v ^^.^r%^ * FS7. These are used to adjust for differences in timing 
processed by circuits ^^^^^ ^en between the fusion prc**ssing format and the input- 
circuits, producing W0) which is stored in FS5. When £ £ fe due to ^ 
this operation is^mpleted, the second i Process ; pass for ^ TOCess ^ g of ^ pyramid levels 
(level 1) is started, where the images G<<1) and G^l) * ^ Another deference may be that the image 
(from FS3 andFS4) are filtered by circuits PI and M format , whi i e the images during the 
and stored in FS3 and FS4 ^ubsampled form G«(2) computed using a progressive 
and G*(2). At the same time, G«(l) and Grfl) are pro- ^ 7 V 

cessed by circuits P3 through P7 and the four fusion mQ 9 fe a ^ ming diagram which shows when the 
circuits, producing I*(l) which is stored in FS5. This vark>us pyramid levels may be computed in 
procedure is repeated until the required number of 35 a system with I/O frame stores, assuming interlace I/O 
pyramid levels are computed. During the last pass (eg. for ft of images. The first time line is for sue- 
level k) the processing is different Then the processing ccss j ve interlaced frames having even and odd fields 
only involves reading G<fls) and G*00 from FS3 and ^ ^ md ^ spaced apart t, y the vertical blanking 
FS4, fusing the images with a different function, and mterva i. The second time line shows the pyramid con- 
storing the result Gc(k) in FS5. This is the remainder or ^ gabion G f the zeroth level gaussian of the pyramid for 
DC component, of the Laplacian pyramid In the 3rd frame which can begin as soon as the odd field 
FIG. 5, this is schematically shown as bypass paths G f the frame is received. The computation of all levels 
around P3 through P7 and the fusion "circuits. The fu- of the 3rd frame pyramid must be completed before the 
sion of the remainders (dc-fusion in FIG. 5) may be as pyramid construction of the zeroth level Gaussian of 
simple as computing the average 45 the 4th frame begins. The third time line shows the 
Many of the processing functions involve 2D filters. pyramid composite image reconstruction for the 3rd 
With 2D filters, the processing at the edge of the image frame which begins at the same time that pyramid con- 
may be undetermined. For large images this may not be struction of the 4th frame begins. The formatted output 
a problem since discarding the edge data has little effect 0 f the 3rd frame begins at the same time that the 5th 
on the totality of the image However, when construct- 50 frame is received. Thus the entire cycle for a given 
ing and reconstructing pyramids, the processing at the frame can be accomplished in two frame times. If the 
edges of the lower resolution levels of the pyramid clock rate of the processing needs to be slightly higher 
affects the final reconstructed image to a significant than the clock rate of the I/O, then first-in, first-out 
extent. For example if five levels of the pyramid are buffers may be used in combination with the I/O frame 
used, a border of 64 pixels around the image may not be 55 stores FS1, FS2 and FSX An alternative to increasing 
reconstructed properly, if the computations at the edges the clock for the fusion processing is to reduce the 
of the images are not performed correctly. The recon- image size on which the fusion processing is being per- 
struction of a pixel includes information from the four formed. 

pixels adjacent to the pixel of interest in a row. For an Many of the processing functions in the implementa- 
edge pixel, two of these adjacent pixels are missing and 60 don (PI, P2, P3, P4, P8, and P9) are 2D filters with 5X5 

for the next adjacent pixel in the row, one of the adja- taps (PI, P2, P8, P9) or 3x3 taps (P3, P4). "Spread tap" 

cent pixels is missing. The simplest way to correct for versions of these filters used in double density pyramid 

this is to insert a constant value in to matrix of pixel construction can also be used. All of these are effi- 

values. Alternatively, the value of the missing pixels are ciently implemented using the PYR-1 circuit described 
set equal to the value of the pixel of interest. A pre- 65 by van der Wal in U.S. patent application Ser. No. 

ferred way to implement the edge processing would be 07/805149 filed Dec. 11, 1991, incorporated herein by 

to reflect or extrapolate the pixel data at the edges of the reference, and in the Workshop For Machine Vision, 

images. For example, at the left edge of an image the Paris, December, 1991 and generally m U.S. Pat No. 
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4,703,514, and sold by the David Sarnoff Research inspection where images taken under differing ffiumina- 
Center, Inc., Princeton, NJ. 08540. t The PYR-1 circuit tion and camera settings (speed, iris, etc.) are combined 
can implement all the : ie<rmre3 'filters, includes the re- to eUmmaie shadows and WgMightsyand dynamic range 1 
quired horizontal line delays, the appropriate adders compression for image display where a large dynamic 
and multiplexing functions, and automatic border con- 5 range image (e.g., a 12 bit medical image) and a display 
trol of the images. Other 2D filter circuits are available that can render only a limited dynamic range (e.g., CRT 
commercially, but require significant additional cir- 0 r LCD) first generate a set of images that represent 
cuitry to incorporate all the functions to implement the limited ranges of the original then combine these images 
methods disclosed here. through fusion for final display. 

An example of a composite image formed from visi- 10 It is to be understood that the apparatus and method 
ble and infrared source images is shown in FIG. 10. The of operation taught herein are illustrative of the inven- 
source image from a standard visible light camera is tf on . Modifications may readily be devised by those 
shown in FIG. 10(a), and the source image from an skilled in the art without departing from the spirit or 
infrared cameras is shown in FIG. 10{b). The fused scope of the invention. The method disclosed here 
image, shown in FIG. 10(c), was obtained using the 15 made ^ Q f edge-like pattern elements of many scales 
gradient pyramid and the saliency and combination and orientations as the local scene features to be used in 
rules outlined above. a composite image. Other directionally sensitive tech- 

The gradient image pyramid provides an effective ' j^qaes for measuring features in an image can also be 
framework for pattern-selective image fusion. Advan- }lst( ^ m the method of the invention. It is also under- 
tages of the pyramid transform for fusion include the 20 stooc j methods other than pyramid processing 
fact that it decomposes-images into regular arrays of methods and means for providing images of different 
edge-like component patterns (basis functions) through scales can be used, 
simple filter operations, and the fact that reconstruction 

blends pattern elements over scales, orientations, and APPENDIX 1 

positions in a seamless manner, avoiding artifacts. These 25 jhc Gradient Pyramid Transform 

amid * applying a gradient operator to each level of its Gauss- 
Our results have shown that fusion within a gradient iau pyramid representation. The image can be corn- 
pyramid is remarkable effective over a wide range of 30 pletely represented by a set of four such gradient 
viewing conditions. A particularly telling set of tests pyramids, one each for derivatives m horizontal, verti- 
involved fusion of extended video sequences containing cal, and the two diagonal directions. . 
objects in motion. In these tests we compared gradient Let G* be the k* level for the Gaussian pyramid for 
pyramid with Laplacian pyramid image fusion. Results I- Then Go(ij)=I(ij) and, for k>0, G*=[w*G*_ih. 
with Laplacian-based fusion tended to have visible, 35 Here w is the generating kernel, and the notation [ . . . 
dynamically changing, artifacts along high contrast h indicates that the image array m brackets is subsam- 
contours. Results with the gradient pyramid were pled (down sampled) by 2 m both the horizontal and 
largely free of such artifacts. vertical directions. ■ 

The observed difference- in performance can be at- D*/ is obtained from G* through convolution with a 

tributed to several factors. Most important is the fact 40 gradient filter d/: 
that the gradient representation has a local maximum in 

value at the location of an edge in the scene, while the . Z, «** -IG *+ 

Laplacian representation has zero value at the edge and . . f «„, ow «h o«h th* irt 

large values either side of the edge. Since the amplitude where D tf is Uie k* 

of the sample value has been used as the measure of 45 option and w is a five by five binomial filter de- 

saliency in the present examples, edge features are more scribed above. . . 

reliably captureTwith the ^adient pyramid than with A method for reconstructing an image from a ; gradi- 
the Laplacian. Furthermore, noise £ the video causes «t pyramid representation comprises the steps of: 
selection at some points in the scene to switch from one « converting each gradient pyramid level D*/ to a 
source image to \he other from frame to frame. The 50 corresponds second denvative pyramid (or on- 
resulting flicker is more visible in Laplacian than in *n*d Up^^Jg*^ ■ i «ood appli- 
gradient based fusion. Humans are most sensitive to cation of the gradient filter Lw*= -{d/WtfKtf-. 
temporal flicker of patterns with low spatial frequency (b) summing the oriented Laplacian pyramids to form 
content Hie gradieit-of-Gaussian basis ftmctions of the an FSD (nUter-subtract-decunate) Uiplacian 
gradient pyramid have a high band-pass characteristic 55 pyramid, L*=S/=£ L*/; 

with significantly less signal energy at low spatial fre- (c) converting the FSD Laplacian L* to an reduce- 

quencies than the Gaussian-like basis functions of the expand (RE) Laplacian U=U+w*L* through a 

Laplacian pyramid. convolution as disclosed by P. Burt, "Fast 

Applications of the image fusion techniques disclosed Filter Transforms For Image Processing , m Mul- 

herein include surveillance using images from multiple 60 tiresolution Image Processing And Analysis, vol- 

sensor types or spectral bands such as visible and IR ume 16, pages 20-51, 1981; and 

cameras; vehicle guidance using multiple sensor types (d) obtaining the reconstructed Gaussian G from the 

(visible, IR, . . . ) as an aid to a driver or pilot at night Reduce-Expand Laplacian through an interpolate 

or in bad weather, combining images taken with a cam- and add procedure using all levels of the Reduce- 

era's focus changed from image to image, to achieve a 65 Expand Laplacian, as well as the top level of the 

composite with an extended depth of field; video special Gaussian: Gn=Gn and for k<N, 

effects using multiple images of difference scenes in GA=Ljfc+4w*[G* + i]^2. Where the notation [ . . . 

order to obtain artistic or other special effects; industrial ]#2 indicates that the image array m brackets is up 
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sampled by inserting n— 1 zero valued rows and bit digital signal containing at least eight data bits and 
columns between each row and column of the two timing bits. 

image:* * """" " *' 1 : ~ ! ' r " i ;v: • The two timing bits convey respective timing signals. ' 

Iterative application of this procedure yields G(0), One signal, HA, is in a logic high state when the data in 
the reconstructed version of the original image G(0). 5 a line is valid (i.e. during the active picture interval) and 
APPFNDTX i ™ a ' og * c l° w statc otn erwise C IC - during the horizontal 

blanking interval). The other signal, VA, is in a logic 
The Pyramid Circuit high state when the data in a field is valid and in a logic 

The pyramid circuit accepts up to three digitized jow state otherwise (i.e. during the vertical blanking 
input signals and provides up to two output signals. The 10 interval). 

input and output data channels incorporate timing sig- The circuit shown in FIG. 11 includes five principal 
nals which control the pipelined processing. These rim- elements: an input arithmetic and logic unit (ALU) 104, 
ing signals are automatically adjusted to the processing a filter 110 (shown within the dashed line in FIG. 1), a 
delay of the circuit, allowing for automatic delay con- multiplexer 120, a clip processor 122 and an output 
trol in pipelined systems. The effective lengths of the 15 ALU 124. Signals INI and IN2, equally delayed by an 
horizontal delay lines used to implement a two-dimen- input delay element 102 are combined by the ALU 104 
sional filter are controlled by the riming signals and, to generate a signal, IN, for application to the filter 110. 
thus, do not need to be programmed. This circuit can This signal may be one of the signals INI or IN2, or it 
accept and process signals having continuously variable may be their sum (IN1+IN2) or their difference (INI- 
horizontal and vertical blanking times. The circuit in- 20 — IN2>. 

eludes programmable edge control which can be used The filter 110 processes the signal provided by the 
to separately extend the data at all edges of the image by ALU 104 through a two dimensional filter which may 
two or four pixels. be configured to have between one and five taps in each 

The digital data signal may be generated from a sen- 0 f ^ two dimensions. The filter includes a vertical edge 
sor or from an analog to digital converter. For an image 25 processor 1M ^ a horizontal edge processor 108 
obtained from a video camera, the horizontal and verti- which ^ low a var i e ty of different types of pixel values 
cal sync signals provide the timing signals. These sig- to ^ ^ surr0 unding the actual image data, 

nals are digitized in an auxiliary device such as a Among these are a constant value or a repeat of the first 
counter and then combined with the digital data agnd Qr last horizontal or vertical line. The processor 108 
to produce the input signal Alternatively the digital 30 rocesses . ^ ■ ^ s to effectively add fines of 
data signal may be generated by a frame store in which hels to ^ at the top and bottom edges, 

case the tuning signal is added by the frame store or an Iq ft ^ ^ ^ ncert ^ three-state gates 

auxiliary device. U2a-112d and a memory 114 to implement a tapped 

The two^imensional fflter uaed in the circuit has a ^ provide four line^elaVed 

separable kernel; it can be treated as a combination of 35 ■ . , J, . ^r.i^ 

serrate horizontal and vertical filters. The five vertical ™& to the vertical filter portion of the convo- 

and five horizontal filter coefficients are programmable lution processor u^^i^^ 
within a limited set of values and may be either symmet- u memo fy P r ™?f a J 0 ^?' eight-line dday for 
ric or antisymmetric. In addition, the filter may be con- *e vertical portion of the tw^dn^onal filter. The 
figured to have either an odd or an even number of taps. 40 delayed hues are combined both verbcaUy and horizon- 

The circuit has two parallel paths which may be used m theconvolution processor 118 to complete the 

to simultaneously calculate a Gaussian low-pass filtered f *er «° The output si^al provided by the filter 110 s 
image and a Laplacian function (the difference of the processed by clip processor 122 which performs round- 
input image and the Gaussian) of the input image. The m B and scaling for smgle preasicm signal and cwnbmes 
two parallel paths are also used for computing the in- 45 ^ flltcred data more significant bit (MSB) post- 
verse pyramid transforms. tions with filtered data representing the less significant 

Multiple pyramid circuits may be cascaded to imple- bit (LSB) portions, provided via input IN3, to generate 
ment multistep filters. Specially programmable delays double-precision output signals, 
and I/O functions allow a relatively large number of The output signal of the processor 122 or the output 
possible configurations to be implemented without ex- 50 signal of the ALU 104, as processed by the processor 
ternal circuitry such as extra delay lines or timing cir- 106, may be selected as the output signal OUTL The 
cuitry . The circuit may be programmed to operate in a output signal OUT2 may be the output signal of the clip 
"spread-tap" mode. This causes the five-by-five tap processor 122 or the output signal of the multiplexer 120 
filter to expand to an nine-by-nine tap filter by effec- or the output of the ALU 124 combining these two 
tively inserting zero-valued coefficients between sue- 55 signals. The signal provided by the multiplexer 120 may 
cessive ones of the five horizontal and five vertical be either a constant value, K2, the input signal IN2 or 
coefficients. one °f ^ c delayed horizontal line signals provided by 

The circuit operates on eight-bit input images (signed the horizontal edge processor 108. The multiplexer 120 
or unsigned) and generates a sixteen-bit result Two includes internal compensating delays (not shown) 
circuits can be connected in parallel so that sixteen-bit 60 which align each of the input signals to the signal pro- 
image data can be processed to produce a fall sixteen-bit vided by the clip processor 122. Timing signals for the 
output signal. circuit are generated by timing circuitry 128 which 

FIG. 11 is a block diagram of an exemplary pyramid produces a two-phase memory clock and a system clock 
circuit used for PI, P2, P7 and P9. The circuit is de- signal CK from a two phase input clock signal, CLK 
signed to function as an element of a multiresolution 65 and CLK2. 

filter. The filter can accept up to three input signals, The circuit functions are controlled via control cir- 
IN1, IN2 and IN3, and provide up to two output sig- cuitry 130 which accepts user commands from a control 
nals, OUT1 and OUT2. Each of these signals is a multi- input channel CBUS and provides control signals and 
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data values to the other components of the circuit via an 13. The method of claim 10 wherein the weight as- 
output port CONT. signed to a particular component pattern in each of the 
I c j a j ini :„.»~...rN .„-'„',,„.■. „■•:-... i . •. . . source images is the weighted-root means square of.the.. : 

I. A method for. forming a composite image from N amplitudes of the particular component partem and 
source images where N is greater than one comprising 5 other components within a local neighborhood. 

the steps of: 14 ^ method of claim 1 

(a) decomposing each source image Ijvinto a plurality wherein step (a) further comprises the step of decom- 

L of sets of oriented component patterns Pjv(m, L), posing each source image into a plurality K of 

where m is the number of patterns in each of the L mages Ik of different resolution and then decom- 

sets for the N«* source image; "> posing each of the images Ijrinto a plurality L of 

Co) computing a saliency measure Sjv(m. L) for each **» ° f ° nent fd component patterns Pj/m, K, L* 

component pattern P^m, L); wherem step (b) composes computmg a sabeacy 

(c) selecting component patterns from the component measure S^m, K, L) for each component pattern 

ororlenL compc.eTpa^ P.m. L) of the S^SSSS 

composite image; and _ , . K L) for each component pattern to form a set of 

(d) constructing the composite .mage from the set of ^patterns P<<m, K, L) of the 
oriented component patterns Pc(m, L). ™/ 

2 Then.^ofclaimlfurther^ingthestep - ^^^^L^n^^cann.o, 

^JffiffidS^^ je^SjTS 6 " ° f ° riCnted ^ 

posing each source image I^into a plurality L of sets of 15 ^ ^^^f claim M whe rein the step of de- 
oriented component patterns PMm. L) comprises ap- 2$ TOmposing ^ ch ^age Xjc mto a plurality L of sets of 
plying oriented operators to an image feature at each Qriented ^po^t patterns Pjv(m, K, L) comprises 
picture element of each source image. applying oriented operators to a feature of each picture 

4. The method of claim 3 wherein each oriented com- element of each source image. 

ponent pattern P^m, L) is the gradient of the local 16 mcthod 0 f 15 wherein each oriented 
image intensity at each picture element of each source ^ component pattern PMm, K, L) is the gradient of the 
image in a particular direction. local image intensity at each picture element of each 

5. The method of claim 4 wherein the gradient opera- source image in a particular direction. 

tors are the first derivatives of the image intensity at 17 ^he method of claim 16 wherein the gradient 
each picture element in first and second orthogonal operators are the first derivatives of the image intensity 
directions in each source image. 35 a t each picture element in first and second orthogonal 

6. The method of claim 5 wherein the gradient opera- directions in each source image. 

tors also include the first derivatives of the image inten- jg The method of claim 17 wherein the gradient 
sity in third and fourth directions in each source image operators also include the first derivatives of the image 
which are orthogonal two one another and at a forty intensity in third and fourth directions in each source 
five degree angle to the first direction. 40 image which are orthogonal two one another and at a 

7. The method of claim 3 wherein the step of select- forty five degree angle to the first direction. 

ing component patterns from the source image pattern 19. The method of claim 14 wherein the step of select- 
sets P^(m, L) for each value of m and L comprises ing component patterns from the source image pattern 
selecting the component pattern having the greatest sets PM m > L) for each value of m and L comprises 
saliency. 45 selecting a linear combination of the amplitudes of the 

8. The method of claim 7 wherein the step of select- component patterns for each image. 

ing component patterns from the source image pattern 20. The method of claim 19 wherein the linear combi- 
sets PMm, L) for each value of m and L comprises nation of the amplitudes of the component patterns for 
selecting the component pattern having the amplitude each image is a weighted average of the amplitudes of 
with the greatest absolute magnitude. 50 the component patterns of the source images. 

9. The method of claim 3 wherein the step of select- 21. The method of claim 20 wherein the weight as- 
ing component patterns from the source image pattern signed to a particular component pattern in each of the 
sets PMm* L ) for each value of m and L comprises source images depends upon the normalized saliency of 
selecting a linear combination of the amplitudes of the the particular component pattern. 

component patterns for each image. 55 22. The method of claim 20 wherein the weighted 

10. The method of claim 9 wherein the linear combi- average of the amplitudes of the component patterns is 
nation of the amplitudes of the component patterns for the weighted root mean square of the amplitudes of the 
each image is a weighted average of the amplitudes of particular component pattern and other components 
the component patterns of the source images. within a local neighborhood. 

II. The method of claim 10 wherein the weight as- 60 23. The method of claim 1 wherein the step of decom- 
signed to a particular component pattern in each of the posing each source image I# comprises forming two or 
source images depends upon the normalized saliency of more sets of wavelets oriented in different directions in 
the particular component pattern. each source image. 

12. The method of claim 10 wherein the weighted 24. The method of claim 23 comprising first and sec- 
average of the amplitudes of the component patterns is 65 ond sets of wavelets in first and second orthogonal 
the weighted root mean square of the amplitudes of the directions. 

particular component pattern and other components 25. The method of claim 24 wherein the step of select- 
within a local neighborhood. ing component patterns from the source image pattern 
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sets PMm, L) for each value of m and L comprises sets PM m » L) for each value of m, K and L corn- 
selecting the component pattern having the greatest prises selecting the component pattern having the great- 
'^salienidyr"-^^^"' -^-^ --^^^uczu^.x^,:^ ^ency; ^' - ■ ^.^a-r- ^.^;,v vtr.: ; ;..;v 

26. The method of claim 24 wherein the step of select- 34. The method of claim 33 wherein the step of select- 
ing component patterns from the source image pattern 5 ing component patterns from the source image pattern 
sets PMm, L) for each value of m and L comprises sets PMm, K, L) for each value of m, K and L corn- 
selecting a linear combination of the amplitudes of the prises selecting the component pattern having the am- 
component patterns for each image. plitude with the greatest absolute magnitude. 

27. The method of claim 1 35. The method of claim 33 wherein the step of select- 
wherein step (a) further comprises the step of decom- 10 ing component patterns from the source image pattern 

posing each source image into a plurality K of gets PM m , K, L) for each value of m, K and L corn- 
images Ijc of different resolution and then decom- prises selecting a linear combination of the amplitudes 
posing each of the images Ijc into a plurality L of 0 f the component patterns for each image, 
sets of oriented component patterns PMm, K* L); 36. The method of claim 35 wherein the linear combi- 

wherein step (b) comprises computing a saliency 15 nation of the amplitudes of the component patterns for 
measure SMm, L) for each component pattern each image is a weighted average of the amplitudes of 
PMm, K L) of each image N; the component patterns of the source images. 

wherein step (c) comprises computing component 37 jhe method of claim 36 wherein the step of de- 
patterns from the source image component pattern composing each source image In comprises forming 
sets PMm, K, L) using the saliency measures SMm, 20 two OT more ^ of waV elets oriented in different direc- 
K L) for each component pattern to form a set of tiorls m ^ j mng ^ 

oriented component patterns P<<m, K, L) of the 33 Thc mctnod 0 f 37 comprising first and sec- 
composite image; and ond sets of wavelets in first and second orthogonal 
wherein step (c) comprises constructing the compos- directions, 
ite image from the set of oriented component pat- 25 39 Apparatus for forming a composite image from a 
terns P^m, K, L). plurality of N source images, where N is greater than 

28. A method for forming a composite image from N £ comprising- 

source images where N is greater than one comprising means V fQT decomposing each source image Ijyinto a 
the steps of: plurality L of sets of oriented component patterns 

30 P I wh ere m is the number of patienis in 

of images I* of duTerent resolution and then de- . ^ ' *^ N ^ source image; 

composing each of the images I^into a plurality L , *7 vr c F „ 

of sete of oriented component patterns PMm, K, means for computing 

L), where m is the numbeVof patterns in each of the f ch component pattern PMm, L) connected to the 

L sets for the N* source image; 35 decomposing means; 

(b) computing a saliency measure SMm, K, L) for means for ****** component patterns from the 
each component pattern PMm, £ L) of each component pattern sets PMm, L) using the sahency 
image N at each resolution level K; measures ^ L )* r ** ch c°W<>nent pattern to 

(c) selecting component patterns from the source form a set of oriented component patterns P^L) 
image component pattern sets PMm* K, L) using 40 of the composite image; and 

the sahency measures SMm, K, L) for each compc- means for constructing the composite image from the 
nent pattern to form a set of oriented component . . «lof oriented component patterns P^m, L). 
patterns Pc(m, K, L) of the composite image; and 40 ™* apparatus of chum 39 further comprising 

(d) constructing the composite image from the set of **** f ° r decomposing each source image mto a plural- 
oriented component patterns P<<m, K, L). 45 *y * of images Ijc of different resolution pnor to the 

29. The method of claim 28 wherein the step of de- means for decomposing each source image, 
composing each source image I* into a plurality L of The apparatus of claim 40 wherein the means for 
sets of oriented component patterns PMm, L) comprises decomposing each source image In comprises a electri- 
applying oriented operators to an image feature at each cal circuit for performing two different gradient opera- 
picture element of each source image. 50 tions on each of the source images I*. 

30. The method of claim 29 wherein each oriented The apparatus of claim 39 wherein the means for 
component pattern PMm, L) is the gradient of the local computing a saliency measure SMm, L) for each corn- 
image intensity at each picture element of each source ponent pattern PMm, L) includes electrical circuitry for 
image in a particular direction. calculating the weights for thc different component 

31. The method of claim 30 wherein the gradient 55 patterns. 

operators are the first derivatives of the image intensity 43. The apparatus of claim 39 wherein the means for 
at each picture element in first and second orthogonal computing a saliency measure SMm, L) for each corn- 
directions in each source image. ponent pattern PMm, L) comprises means for calculat- 

32. The method of claim 31 wherein the gradient ing the root mean square weight for the amplitudes 
operators also include the first derivatives of the image 60 different component patterns. 

intensity in third and fourth directions in each source 44. The apparatus of claim 41 wherein the means for 
image which are orthogonal two one another and at a constructing each composite image I at comprises a elec- 
forty five degree angle to the first direction, trical circuit for performing the same two gradient op- 

33. The method of claim 28 wherein the step of select- erations on each set of composite patterns, 
ing component patterns from the source image pattern 65 



* * * ♦ 
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